Internal / Draft / Roadmap / Elements / File Storage

File Storage

Last updated on 15-Mar-2018 by Jakob Jakobsen Boysen
Jakob Jakobsen Boysen

Platform Lead
boysen@scifeon.com
, Thomas P. Boesen
Thomas P. Boesen

Founder
boesen@scifeon.com
On this page

This document describes how Scifeon should handle files of all kinds, except App files - these are handled differently as they are part of a package managed by administrators. Files should be handled differently than simple database records, due to capacity and performance issues.

Scope

Scifeon is not a file storage system like Dropbox or Google Drive, but for convenience it should be possible to upload files, i.e. relate files, to the following entities:

  • Experiments
  • Steps
  • ResultSets

The files are not organized in folder hierarchies, but an experiment can easily show all directly and indirectly related files in a hierarchy mimicking the step hierarchy.

Requirements and specifications

Uploading

  • It should be possible to upload files from the experiment page, either by dragging and dropping files, or by selecting files using a "Browse..." button
    • If the Experiment contains Steps (w/wo ResultSets) show a dialog where the user chooses which entity to upload the file to (attach to)
  • It should be possible to choose an entity to upload the current file to, when using Data Loaders on the Upload Data page
    • If the new Entity set contains one of the following entities, choose that (in prioritized order): Experiment, Step, ResultSet
    • If the new Entity set doesn't contain any of the above, just choose the first of the Entity set

The upload mechanism must show a progress bar, as files can be huge.

Downloading

It should be possible to download files just by clicking links referring to files, e.g. on the experiment page.

Rendering

A standard panel for listing files should be implemented as a plugin. This can be used as renderer on Steps in the Experiment page.

Size limits

It should be possible (through configuration) to set the file size limit for uploads. This limit should be enforced by the UI and the HTTP API endpoints. The host (e.g. IIS) should not set any limits.

File Storage Providers

It should be possible to choose between different file storage providers, depending on the current needs and possibilities. In the first version of this concept, the following File Storage Providers should be supported:

  • FileContent table: this provider is just storing files in the FileContent table in Scifeon. This should be used only for local development and SDK instances
  • Azure Storage: this provider uses Azure Blob Storage as storage

In the next iteration of the concept, the following File Storage Providers would also make sense:

Operations

Each File Storage Provider must support the following operations:

  • Adding files
  • Updating files
  • Removing files
  • Downloading files

Data model

The existing Assay_ResultFile should be ditched completely, Core_File is introduced with a ResultSetID and Position field. The Position field is a free-text field...

The dashed boxes indicates examples of how fields can be used.

The Assay_ResultImage table currently has ImageData and Format fields. These are replaced by the FileID reference. Images can still be shown directly using e.g. 

<img src="{hostname}/api/files/#FileID"/>
Copyhtml

Configuration

The data model indicates that some configuration need to be in place. To avoid too much initial configuration of new or existing instances of Scifeon, we use the sensible default to fallback to using the FileContent provider, if nothing else have been setup. This means if the StorageID is empty, the FileContent provider is used.

appsettings.json should now contain the following section:

"scirexSettings": {
    "storageProvider": "DB"
}
CopyJSON

storageProvider can be one of the following:

  • DB: Persisted in Core_FileContent
  • Azure: Persisted in an Azure Blob Storage
  • LocalFile: Persisted in a simple folder on the hosting server
  • SMB: Persisted on a file share using the CIFS protocol

Configuration JSON, Azure

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-dotnet-how-to-use-blobs

https://docs.microsoft.com/en-us/azure/storage/common/storage-use-emulator

{
    "accountName": "",
    "accountKey": ""
}
CopyJSON

Used to construct the following connection string: DefaultEndpointsProtocol=https;AccountName={accountName};AccountKey={accountKey}

Configuration JSON, LocalFile

{
    "path": "e:/path/to/files"
}
CopyJSON

Configuration JSON, SMB

http://sharpcifsstd.dobes.jp/

{
    "host": "",
    "sharename": "",
    "path": "path/to/files/on/sharename",
    "port": "",
    "username": "",
    "password": "",
}
CopyJSON

Used to construct the following connection string: smb://{userName}:{password}@{hostname}/{sharename}/{path}/

HTTP API

Access to the file storage should go through dedicated, agnostic API endpoints, as the underlying implementation should be hidden from the user. The implementation is accessed through the StorageProviderID field of the File. The following API endpoints should be supported:

  • POST /api/files:
  • PUT /api/files/#ID:
  • DELETE /api/files/#ID:
  • GET /api/files/#ID: returns the file in the body and with the correct headers
  • GET /api/files/#ID/meta: returns the meta data of the file
On this page